Commentary for special issue of journal of memory and language: Generalizability theory analysis for psycholinguistic applications

We propose that the continuing controversies over the use of quasi-F-ratios in psycholinguistic research might be circumvented, if not resolved, by the judicious application of Generalizability Theory (GT) analyses. We argue that GT is a logical extension of the basic rationale behind repeated measures Analysis of Variance (ANOVA) and the variance components model upon which GT is ultimately based and upon which the entire logic of the F-ratio (quasi or otherwise) rests. GT is especially useful in psycholinguistics research because it affords one the opportunity to assess generalizability across multiple dimensions within the same model, such as individual subjects as well as varying conditions of prime and target words. We will provide an illustrative example of GT based on Forster’s (2007) replication of Davis and Lupker’s (2006) study in which they tested the effects of frequency discrepancies in target and prime words across individual subjects under varying combinations of frequency and prime. 2008 Elsevier Inc. All rights reserved. With advance apologies to the statistically more sophisticated in the readership, we approach this matter didactically by briefly reviewing a few of the basic concepts of Analysis of Variance (ANOVA) so as to develop a common symbolic lexicon with which to build a bridge to Generalizability Theory (GT) Analysis. By building on the wellknown, we hope to shed some light upon the less familiar and render it more accessible to a wider readership. The fundamental logic of the F-ratio We start with the basic theoretical formula for the F-ratio: F 1⁄4 ðrt þ reÞ=re ð1Þ Because any estimated treatment effect (e.g., variance between experimental groups) contains some ‘‘error” (e.g., sampling error in a between-groups design), to test the ‘‘true” estimated treatment effects it is necessary to divide this numerator by an appropriate denominator that is a purer estimate of the so-called ‘‘error” (e.g., variance within experimental groups). The whole point of the F-ratio was to determine whether the hypothetical variance component representing the ‘‘true” treatment effect (rt ), without the ever-present ‘‘error” component (re), is (or is not) equal to zero. Because, in Sir Ronald Fisher’s time, we had no way to directly determine this, we instead constructed the F-ratio to test that ‘‘null hypothesis”. If it was indeed the case that the hypothetical variance component for treatment equaled zero (rt = 0), then the expected value of the F-ratio would be 1.0 because it would reduce to simply re/re . What is typically meant by an ‘‘error” term in ANOVA is more properly referred to as a random effect. The levels of a random effect are conceptualized as being randomly selected from a population and are thus in principle exchangeable with another sample from the same population. The goal of using random effects is to be able to generalize the results to that population. In conventional ANOVA, individuals (variously referred to persons, sub0749-596X/$ see front matter 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.jml.2008.07.004 * Corresponding author. Fax: +1 520 621 9306. E-mail address: [email protected] (A.J. Figueredo). Journal of Memory and Language 59 (2008) 426–433